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Executive  Summary 


Motivation  and  Context 

One  of  the  key  elements  of  the  SERC's  research  strategy  is  transforming  the  practice  of 
systems  engineering  and  associated  management  practices-  "SE  and  Management 
Transformation  (SEMT)."  The  Grand  Challenge  goal  for  SEMT  is  to  transform  the  DoD 
community's  current  systems  engineering  and  management  methods,  processes,  and 
tools  (MPTs)  and  practices  away  from  sequential,  single  stovepipe  system,  hardware-first 
, document-driven,  point-  solution,  acquisition-oriented  approaches;  and  toward 
concurrent,  portfolio  and  enterprise-  oriented,  hardware-software-human  engineered, 
model-driven,  set-based,  full  life  cycle  approaches. 

These  will  enable  much  more  rapid,  concurrent,  flexible,  scalable  definition  and  analysis 
of  the  increasingly  complex,  dynamic,  multi-stakeholder,  cyber-physical-human  DoD 
systems  of  the  future. Four  elements  of  the  research  strategy  for  SE  Transformation  are 
the  following: 

1.  Make  Smart  Trades  Quickly:  Develop  MPTs  to  enable  stakeholders  to  be  able  to 
understand  and  visualize  the  tradespace  and  make  smart  decisions  quickly  that  take 
into  account  how  the  many  characteristics  and  functions  of  systems  impact  each 
other 

2.  Rapidly  Conceive  of  Systems:  Develop  MPTs  that  allow  multi-discipline 
stakeholders  to  quickly  develop  alternative  system  concepts  and  evaluate  them 
for  their  effectiveness  and  practicality 

3.  Balance  Agility,  Assurance,  and  Affordability:  Develop  SE  MPTs  that  work  with 
high  assurance  in  the  face  of  high  uncertainty  and  rapid  change  in  mission, 
requirements,  technology,  and  otherfactors  to  allow  systems  to  be  rapidly  and 
cost-effectively  acquired  and  responsive  to  both  anticipated  and  unanticipated 
changes  inthefield 

4.  Align  with  Engineered  Resilient  Systems  (ERS):  Align  research  to  leverage  DoD's 
ERS  strategic  research  initiative  and  contribute  to  it;  e.g.,  ERS  efforts  to  define 
new  approaches  to  tradespace  analysis. 
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"Systems"  covers  the  full  range  of  DoD  systems  of  interest  from  components  such  as 
sensors  and  effectors  to  systems  of  systems  that  are  full  or  parts  of  net-centric  systems 
of  systems  and  enterprises. "Effectiveness"  covers  the  full  range  of  needed  System 
Qualities  (SQs)  such  as  reliability,  availability,  maintainability,  safety,  security, 
performance,  usability,  scalability,  interoperability,  speed,  versatility,  flexibility  and 
adaptability  along  with  composite  attributes  such  as  resilience,  affordability,  and 
suitability  or  mission  effectiveness.  "Cost"  covers  thefull  range  of  needed  resources, 
including  present  and  future  dollars,  calendar  time,  critical  skills,  and  critical  material 
resources. 

The  primary  focus  of  RT-160,  Phase  5  of  the  System  Qualities  Ontology,  Trades  pace  and 
Affordability  (SQOTA)  project  is  on  strategy  3,  although  its  capabilities  also  support 
strategies  1,  2,  and  4.  It  particularly  focuses  on  the  tradespace  among  a  system's  qualities, 
also  called  non-functional  requirements  or  system  ilities.  The  SQs  differ  from  functional 
requirements  in  that  they  are  systemwide  properties  that  specify  how  well  the  system 
should  perform,  as  compared  to  functions  that  specify  what  the  system  should  perform. 
Adding  a  functional  requirement  to  a  system's  specification  tends  to  have  an  incremental, 
additive  effect  on  the  system's  cost  and  schedule.  Adding  an  SQrequirementto  a  system's 
specification  tends  to  have  a  systemwide,  multiplicative  effect  on  the  system's  cost  and 
schedule.  Also,  SQs  are  harderto  specify  and  evaluate,  as  their  values  vary  with  variations 
inthe  system's  environment  and  operational  scenarios. 

Further,  the  satisfaction  of  their  specifications  is  much  harder  to  verify  than  placing  an  X 
in  a  functional  traceability  matrix,  as  the  verification  traces  to  the  entire  set  of  system 
functions.  It  also  requires  considerable  effort  in  analysis  across  a  range  of  environments 
and  operational  scenarios.  As  a  result,  it  is  not  surprising  that  problems  in  satisfying  SQ 
requirements  are  the  source  of  many  DoD  acquisition  program  cost  and  schedule 
overruns.  Also,  with  some  exceptions  such  as  pure  physical  systems  and  pure  software 
systems,  there  is  little  technology  in  the  form  of  scalable  methods,  processes,  and  tools 
(MPTs)  for  evaluating  the  satisfaction  of  multiple-SQ  requirements  and  their  associated 
tradespacesfor  complex  cyber-physical-human  systems. 

The  increasingly  critical  DoD  need  for  such  capabilities  has  been  identified  in  several 
recent  studies  and  initiatives  such  as  the  AFRL  "Technology  Florizons"  report  (Dahm, 
2010),  the  National  Research  Council's  "Critical  Code"  Report  (NRC,  2010),  the  SERC 
"Systems  2020"  Report  (SERC,  2010),  the  "Manual  for  the  Operation  of  the  Joint 
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Capabilities  Integration  and  Development  System"  (JROC,  2012),  and  the  DoD  "Engineered 
Resilient  Systems  (ERS)  Roadmap"  (Holland,  2012).  The  particular  need  for  Affordability 
has  been  emphasized  in  several  USD(AT&L)  and  DepSecDef  "Better  Buying  Power" 
memoranda  BBP  l.Oand 2.0 (Carter  etal.,  2010-2013)  andthe  recent  BBP  3.0  White  Paper 
(Kendall,  2014). 


SQOTA  Phase  5  context:  Summary  of  Phases  1  through  4 

For  SQOTA  Phase  5  context,  short  summaries  of  SQOTA  Phases  1  through  4  are  provided 
next. 


Phase  1  Objectives,  Approach  and  Results 

The  major  objectives  of  the  initial  5-month  Phase  lactivity  were  to  lay  strong 
foundations  for  SQOTA  Phase  2,  including  knowledge  of  Department  of  Defense  (DoD) 
SQ  priorities;  foundations  and  frameworks  for  SQtradespace  analysis;  extension  and 
tailoring  of  existing  SQOTA  methods,  processes,  and  tools  (MPTs);  and  exploration  of 
candidate  Phase  2  pilot  organizations  for  ITAP  MPTs. 

Four  activities  were  pursued  in  achievingthese  objectives: 

1.  SQ  Definitions  and  Relationships.  Phase  lincluded  a  discovery  activity  to  identify 

and  analyze  DoD  and  other  ility  definitions  and  relationships,  and  to  propose  a 
draft  set  of  DoD-oriented  working  definitions  and  relationships  for  the  project. 

2.  SQ  Foundations  and  Frameworks.  This  effort  helped  to  build  SQOTA 
foundations  by  elaborating  key  frameworks  (process-based,  architecture- 
based,  means-ends  based,  value-basedL  anticipating  further  subsequent 
elaboration  via  community  efforts. 

3.  SQ-Oriented  tool  demos  and  extension  plans.  This  effort  created  initial 
demonstration  capabilities  from  strong  existing  SERC  SQ  analysis  toolsets  and 
explored  piloting  by  user  organizations  inthe  DoDServices. 

4.  Program  management  and  community  building.  This  effort  included 
coordinating  efforts  with  complementary  initiatives  in  the  DoD  ERS,  and 
counterpart  working  groups  inthe  International  Council  for  Systems  Engineering 
(INCOSE),  the  Military  Operations  Research  Society  (MORS),  and  the  National 
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Defense  industry  Association  (NDIA). 


The  Phase  lresults  for  activities  land  2  included  initial  top-level  sets  of  views  relevant  to 
SQ  tradespace  and  affordability  analysis  that  provided  an  initial  common  framework  for 
reasoning  about  SQs,  similar  in  intent  to  the  various  views  provided  by  SysMLfor  product 
architectures  and  DoDAF  for  operational  and  architectural  views.  The  views  included 
definitions,  stakeholder  value-based  and  change-oriented  views,  views  of  ility  synergies 
and  conflicts  resultingfrom  ility  achievement  strategies,  and  a  representation  scheme  and 
support  system  for  view  construction  and  analysis. 

Phase  lalso  determined  that  strongtradespace  capabilities  were  beingdeveloped  forthe 
tradespace  analysis  of  physical  systems.  However,  based  on  sources  such  as  the  JCIDS  survey 
of  combat  commanders'  tradespace  needs,  itfound  that  major  gaps  existed  between 
commanders' SQ  tradespace  needs  and  available  capabilities  for  current  and  future 
cyber- physical-human  systems.The  SERC  also  characterized  the  benefits  and  limitations 
of  using  existing  tools  to  address  SQ  tradespace  issues,  via  collaboration  with  other 
leading  organizations  in  the  DoD  ERS  tradespace  area,  such  as  the  Army  Engineer  Research 
and  Development  Center  (ERDC)  and  TARDEC  organizations,  NAVSEA,  the  USAF  Space  and 
Missile  Systems  Command;  DoD  FFRDCs  such  as  Aerospace,  Mitre,  and  the  Software 
Engineering  Institute;  and  Air  Force  and  Navy  participants  via  the  SERC  Service  academies 
AFITand  NPS. 


Phase  2  Objectives,  Approach  and  Results 

As  a  result,  the  focus  of  Phase  2  was  to  strengthen  the  conceptual  frameworks 
underlyingSQ.  tradespace  and  affordability  analysis,  and  to  apply  the  methods  and  tools 
identified  and  extended  in  Phase  Ion  problems  relevantto  DoD,  usingthe  information 
availablefrom  development  of  a  large  weapon  systems  and  large  automated  information 
systems.  The  SERC  worked  with  system  developers  directly  and  via  participation  and 
leadership  in  Government  and  industry  working  groups  in  such  organizations  as  INCOSE, 
NDIA,  and  the  Army-led  Practical  Systems  and  Software  Measurement  organization,  to 
gain  a  deeper  shared  understanding  of  the  strengths  and  limitations  of  the  tradespace 
tools  and  methods  developed  under  Phase  1  and  elsewhere. 

Task  1: /TAP  Foundations  and  Frameworks.  Phase  2  activities  expanded  the  set  of  SQs 
represented  in  the  tradespace,  organized  them  into  a  more  orthogonal  value-based, 
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means-  ends  hierarchy,  obtained  initial  results  in  identifying  and  quantifying  the 
synergies  and  conflicts  resulting  from  strategies  to  optimize  individual  SQs,  and 
developed  prototype  tools  for  representing  and  applyingthe  results. 

Task  2.  iTAP  Methods  and  Tools  Piloting  and  Refinement.  The  SQ-oriented  tool  demos 
performed  in  Phase  lalso  led  to  Phase  2  interactions  with  DoD  organizations,  particularly 
TARDEC  and  NAVSEA,  interested  in  their  applicability  in  enhancing  their  systems  engineering 
capabilities.  These  interactions  led  to  refinements  of  existing  methods  and  tools  to  address 
set-  based  vs.  point  design  of  ground  vehicles  and  ships,  and  on  extensions  from  physical 
systems  to  cyber-physical-human  systems  and  to  affordability  analysis.  Further 
interactions  leading  to  piloting  engagements  included  AFIT's  use  of  the  CEVLCC  lifecycle  cost 
model  and  related  T-X  Training  System  Tradespace  Analyses.  The  pilot  program  involved 
advanced  pilot  training  aircraft,  simulators  and  course  instructional  elements.  Its  pilot 
organizations  were  the  Air  Force  life  Cycle  Management  Center  and  the  Air  Education  and 
Training  Command.  GTRI's 

Framework  for  Assessing  Cost  and  Technology  (FACT)  was  extended  beyond  its  initial  support 
of  USMC,  and  attracted  several  Army  and  Navy  programs  interested  in  piloting,  extending, 
and  tailoring  its  capabilities  to  other  domains. 

Task  3.  Next-Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles.  A  third  area 
of  engagement  starting  from  exploratory  discussions  in  Phase  lwas  a  new  task  to 
develop  Next- Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles,  initially  for 
the  space  domain,  based  on  discussions  and  initial  support  from  the  USAF  Space  and 
Missile  Systems  Center  (SMC).  Phase  2  work  on  this  topic  involved  several  meetings 
with  SMC  and  the  Aerospace  Corp.  with  USC  and  NPS  to  set  context  and  initial  priorities. 
These  included  addressal  of  future  cost  estimation  challenges  identified  in  the  SERC  RT-6 
Software  Cost  Estimation  Metrics  Manual  developed  for  the  Air  Force  Cost  Analysis 
Agency,  and  prioritization  of  research  efforts  based  on  strength  of  DoD  needs  and 
availability  of  DoD-relevant  data.  Exploratory  activities  were  pursued  with  respect  to  a 
seeping  of  full-coverage  of  space  system  flight,  ground,  and  launch  systems;  hardware, 
software  and  labor  costs;  and  system  definition,  development,  operations,  and  support 
costs,  along  with  explorations  of  sources  of  data  for  calibratingthe  models. 
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Phase  3  Objectives,  Approach  and  Results 


TaskliSQFoundations  and  Frameworks.  MIT's  Phase  2  research  refined  a  SQs  semantic  basis 
for  change-related  SQs.  and  developed  prototype  tools  for  formal  analysis  of  the  results. 
Phase  3  extended  the  SQs  semantic  basis  for  change-related  SQs,  resulting  from 
continuing  literature  review  of  SQs,  collaborative  work  on  formalization  of  the  basis,  and 
experience  in  applyingthe  basis  in  historical  cases.  Progress  and  adjustments  tothe  basis 
have  been  made  as  a  result  offeedbackfrom  otheracademic  researchers,  and  specifically 
in  MIT-  UVa  collaboration  in  their  efforts  on  formalization  and  development  of  a  REST 
(representational  state  transfer)  web-based  service  implementation.  This  resulted  in  an 
expanded  and  more  explicit  representation  for  the  semantic  basis,  as  well  as  motivating 
the  need  to  create  a  translation  layerfor  practical  use  ofthe  basis.  Phase  3  also  refined 
the  SQ  definitions,  reviewed  existing  SQ  definition  standards,  developed  an  initial  SQs 
ontology  reflectingthe  realitythatthe  ilities  have  multiple  definitions  varying  by  domain, 
and  multiple  values  varying  by  system  state,  processes,  and  relations  with  other  ility 
levels.  Phase  3  also  expanded  the  initial  4x4  synergies  and  conflicts  matrix  into  a  full  7x7 
inter-ility-class  synergies  and  conflicts  matrix,  and  7smaller  intra-ility-classsynergiesand 
conflicts  matrices. 

Task  2:  SQ-Oriented  tool  demos  and  extension  plans.  Phase  2  effort  created  initial 
demonstration  capabilities  from  strong  existing  SQ  analysis  toolsets  and  explored 
piloting  by  user  organizations,  via  collaboration  with  other  leading  organizations  in  the 
DoD  ERS 

trades  pace  area.  Phase  3  broadened  and  deepened  these  initial  contacts,  including  with 
such  organizations  as  the  Army  Engineer  Research  and  Development  Center  (ERDC)  and 
TARDEC  organizations,  NAVSEA,  the  USAF  Space  and  Missile  Systems  Command;  DoD  FFRDCs 
such  as  Aerospace,  Mitre,  and  the  Software  Engineering  Institute;  and  Air  Force  and  Navy 
participants  via  the  SERC  Service  academies  AFIT  and  NPS.  In  particular,  WSU  and  PSU 
advanced  the  SQOTA  coordination  with  the  ERS  NAVSEA  group,  working  with  them  to  define 
the  specific  tradespace  approaches  and  priorities  for  enhanced  set-based  design  for  ERS, 
that  will  complement  and  extend  the  tool  and  procedures  they  have  been  using.  TARDEC 
was  actively  engaged  as  a  partner  for  co-development,  piloting  and  transition  into  use.  The 
GTRI  FACT-related  capabilities  were  strongly  co-funded  and  enhanced  by  and  for  the  Army 
Engineer  R&D  Center  (ERDC),  other  Army,  Navy,  and  further  USMC  programs,  including 
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strengthening  and  extension  of  the  infrastructure  for  supporting  and  extending  the  initial 
FACT  capabilities. 


Task  3:  Next-Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles.  Based  on  the 
exploratory  needs  and  data  assessments  in  Phase  2,  a  Phase  3  workshop  including  Air 
Force,  Navy,  aerospace  industry,  and  SERC  researchers  concluded  thatthere  were  strong 
needsfor  better  estimation  of  operations  and  support  costs,  but  that  the  data  available 
lacked  adequate  cost  driver  information,  except  in  in  the  software  area.  The  workshop 
recommended  that  the  most  promising  initial  areas  to  pursue  would  be  for  software 
development,  systems  engineering,  and  the  use  of  systems  engineering  cost  drivers  to 
improve  estimation  of  system  development  costs.  Further  research  and  workshops 
identified  further  sources  of  data  and  some  shortfalls  in  current  models  in  these  areas, 
and  developed  requirements  and  draft  frameworks  for  the  next-generation  models. 
These  have  been  used  in  Phase4to  develop  and  calibrate  prototype  modelsfor  systems 
and  software  engineering  cost  estimation  models,  and  to  pursue  research  in  the  use  of 
the  systems  engineering  model  to  better  estimate  system  developmentcosts. 


Phase  4  Objectives,  Approach  and  Results 
Taskl.SQ  Foundations  and  Frameworks 


Rather  than  attempt  a  breadth-first  elaboration  of  the  176  SQ  Synergies  and  Conflicts 
strategies  in  the  7x7  matrix,  including  its  ontology  elements  of  Referents,  States, 
Processes,  and  Relations  for  each  strategy,  the  USC  ontology-based  research  id  a  depth- 
first  research  effort  on  a  particular  SQthat  touches  all  of  the  four  major  SQ  categories. 
This  SQis  Maintainability.  It  clearly  drives  Life  Cycle  Efficiency,  as  typically  at  least  75%  of 
a  system's  Total  Cost  of 

Ownership  isspenton  operations  and  maintenance.  Itisoneoftwo  meansfor  achieving 
Changeability,  involving  external  change  vs.  the  internal  change  accomplished  by 
Adaptability.  It  is  clearly  key  to  Dependability,  as  Maintainability  interms  of  MeanTime 
to  Repair  (MTIR)  is  the  key  relation  between  Reliability  in  terms  of  MeanTime  Between 
Failures  (MTBF)  and 

Availability  inthe  relation  Availability=  MTBF /(MTBF  +  MTIR).  And  the  key  systems  aspects 
being  depended-upon  are  primarily  the  components  of  Mission  Effectiveness. 
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This  depth-first  approach  thus  provided  insights  on  the  overall  Product  Quality  ontology 
structure  without  having  to  consider  all  of  the  176  strategies  in  depth.  The  insights 
resulted  in  changes  to  the  SQterminology,  as  shown  inthe  main  description  of  the  Phase 
4  Results.  Examples  are  changing  Resource  Utilization  to  Life  Cycle  Efficiency,  to  be  more 
compatible  with  the  Better  Buying  Power  terminology,  and  changing  Flexibility  to 
Changeability,  to  be  better  aligned  with  the  MIT  Quality  In  Use  ontology  structure. 

The  MIT  Quality  In  Use  ontology  structure  was  refined  to  address  further  semantic  aspects, 
and  requirements  for  a  translation  layerto  facilitate  its  use  were  developed,  as  elaborated 
inthe  main  Phase4results  section.  Similarly,  the  U. Virginia  Phase  4  research  on  formalizing 
both  the  MITand  USContologies  iselaborated  inthe  main  Phase4  results  section.  An  initial 
semantic  diagram  relatingthe  USCand  MITterms  and  relationships  is  also  presented  inthe 
main  Phase4  resultssection. 

Task  2.  SQ-Oriented  tool  demos  and  extension  plans. 


The  USC  depth-first  exploration  of  Maintainability  identified  the  need  for  a  better 
balance  of  attention  during  the  system  acquisition  phase  between  optimizing  on  system 
acquisition  cost-  effectiveness  and  optimizing  on  system  life-cycle  cost-effectiveness, 
particularly  for  software,  due  to  the  major  differences  between  software  and  software 
logistical  aspects.  This  led  to  the  development  of  a  proposed  framework  of 
Maintainability  Readiness  Levels  for  Software  -  Intensive  Systems.  Again,  details  are 
provided  in  the  main  Phase  4  results  section. 

Other  Task  2  Phase  4  Objectives,  Tasks  and  Results  summaries  will  be  provided  later. 


Task  3:  Next-Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles 

The  Task  3  Phase  4  Objectives,  Tasks  and  Results  summaries  will  be  provided  later. 
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Phase  5  Results  Summary 


1.2.1  Task  1:  System  Qualities  (SQ)  Foundations:  SQ  Ontology 

SQ  Ontology  (USC,  AFIT,  MIT,  U. Virginia).  A  major  development  in  2016  was  the 
publication  of  aworkable  ontology  of  the  nature  and  relations  ofthe  SQs.  Itbuilt  upon 
partial  ontologies  by  David  Jacques  and  Erin  Ryan  at  AFIT;  by  Adam  Ross  and  Donna 
Rhodes  at  MIT;  and  Bany  Boehm  and  Jo  Ann  Lane  at  USC,  along  win  an  initial  formal 
definition  ofthe  relations  among  the  SQs  by  Kevin  Sullivan  at  U.  Virginia.  One  part  ofthe 
ontology,  shown  in  Figure  1,  organizes  the  SQs  into  a  class  hierarchy  reflecting  system 
stakeholders'  value  propositions  (Mission  Effectiveness,  Life  Cycle  Efficiency, 
Dependability,  Changeability),  and  the  means  for  satisfying  them.  Other  parts  of  the 
ontology  identify  the  sources  of  variation  in  an  SQ's  numerical  value  with  respect  to 
stakeholder  priorities;  internal  and  external  system  states  and  processes;  and  synergy 
and  conflict  relations  amongthe  SQs. 


Table  1  Upper  Levels  of  Stakeholder  Value-Based  SO  Means-Ends  Hierarchy 


Stakeholder  Value-Based  SO 

Ends 

Contributing  SQ  Means 

Mission  Effectiveness 

Stakeholders-satisfactory  balance  of  Physical  Capability, Cyber  Capability, 

Human  Usability,  Speed,  Endurability,  Maneuverability,  Accuracy,  Impact, 
Scalability,  Versatility ,  Interoperability,  Domain-Specific  Objectives 
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Life  Cycle  Efficiency 

Development  and  Maintenance  Cost,  Duration,  Key  Personnel, Other  Scarce 
Resources; Manufacturability ,  Sustainability 

Dependability 

Reliability,  Maintainability, Availability  ,  Survivability  Robustness,  Graceful 

Degradation,  Security,  Safety 

Changeability 

Maintainability ,  Modifiability ,  Repairability ,  Adaptability 

Composite  SQs 

Affordability 

Mission  Effectiveness,  Life  Cycle  Efficiency 

Resilience 

Dependability,  Changeability 

One  Aspect  of  the  ontology  addressed  in  2016  was  to  identify  maintainability  as  a  means  of 
achieving  several  higher-level  value  propositions.  It  contributes  to  dependability  as  Mean  Time 
to  Repair  (MTTR)  in  the  relationship  between  Reliability  as  Mean  Time  Between  Failures.  This 
(MTBF)  and  Availability  via  the  equation  Availability  =  MTBF  /  (MTBF+MTTR).  It  also  contributes 
to  Changeability  and  Life  Cycle  Efficiency  by  making  repairs  and  change  requests  less  costly. 
Combined  with  the  fact  that  DoD  and  most  other  organizations  spend  about  75%  of  their  budgets 
on  systems  maintenance,  this  led  us  to  do  a  deep  dive  on  maintainability  to  determine  the  main 
sources  of  systems'  maintenance  costs  and  what  could  be  done  to  reduce  them. 

This  included  data-analytics  and  experimental  studies  of  alternative  automated  (Maintainability 
Index,  Teclmical  Debt)  and  human-assessed  (Software  Understanding)  Maintainability  metrics; 
and  workshops  with  our  industry  and  government  affiliates  to  identify  the  major  causes  of 
Technical  Debt  and  other  maintenance  difficulties.  The  results  of  the  studies  were  that  the 
human-assessed  metrics  were  better  predictors  of  maintenance  effort,  but  that  the  automated 
methods  were  more  efficient,  and  that  the  best  approach  was  to  use  the  automated  methods  to 
identify  the  parts  of  the  software  that  would  most  benefit  from  human  assessment.  The  results 
of  the  workshops  were  that  the  debt  was  technical,  but  its  main  root  causes  were  non-technical. 

A  top-1  0  list  of  the  main  root  causes  follows:  Separate  organizations  and  budgets  for  systems 
and  software  acquisition  and  maintenance;  Over  concern  with  the  Voice  of  the  Customer;  The 
Conspiracy  of  Optimism;  Inadequate  system  engineering  resources;  Hasty  contracting  that 
focuses  on  fixed  operational  requirements;  CAIV-limited  system  requirements;  Brittle,  point- 
solution  architectures;  The  maintainers'  Vicious  Circle;  Stovepipe  systems;  and  Over-extreme 
forms  of  agile  development. 

As  a  way  of  focusing  management  attention  on  the  life  cycle  operational  needs  and  financial 
benefits  of  developing  maintainable  systems,  the  project  has  developed  a  counterpart  to  the 
Technology  Readiness  Level  framework,  called  the  System/Software  Maintainability  Readiness 
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Framework  (SMRF),  shown  in  Table  2.  Several  DoD-community  organizations  are  evaluating  it 
for  use 


Table  2  Software-Intensive  Systems  Maintainability  Readiness  Framework 


Software-Intensive  Systems  Maintainability  Readiness  Framework  (SMRF) 

SMR 

Level 

Op  Con,  Contracting: 

Missions,  Scenarios, 
Resources,  Incentives 

Personnel  Capabilities  and 
Participation 

Enabling  Methods, 

Processes,  and  Tools  (MPTs) 

9 

5  years  of  successful  maintenance 
operations,  including  outcome- 
based  incentives,  adaptation  to 
new  technologies,  missions,  and 

stakeholders 

In  addition,  creating  incentives 
for  continuing  effective 
maintainability, 
performance  on  long-duration 
projects 

Evidence  of  improvements  in 

innovative  O&M  MPTs 
based  on  ongoing  O&M 
experience 

8 

One  year  of  successful 
maintenance  operations, 
including  outcome  based 
incentives,  refinements  of  OpCon. 
Initial  insights  from  maintenance 
data  collection  and  analysis 
(DC&A) 

Stimulating  and  applying  People 
CMM  Level  5  maintainability 
practices  in  continuous 
improvement  and  innovation  in, 
e.g.,  smart  systems,  use  of 
multicore  processors,  and  3-D 
printing 

Evidence  of  MPT  improvements 
based  on  maintenance  DC&A- 

based  ongoing  refinement,  and 
extensions  of  ongoing  evaluation, 

initial  O&M  MPTs. 

7 

System  passes  Maintainability 

Readiness  Review  with  evidence 
of  viable  OpCon,  Contracting, 
Logistics,  Resources,  Incentives, 
personnel  capabilities,  enabling 
MPTs,  outcome-based  incentives 

Achieving  advanced  People  CMM 
Level  4  maintainability 
capabilities  such  as  empowered 
work  groups,  mentoring, 
quantitative  performance 
management  and  competency- 

based  assets 

Advanced,  integrated,  tested,  and 
exercised  fuli-LC  MBS&SE  MPTs 

and  Maintainability-other-SQ 
tradespace  analysis 
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6 

Mostly-elaborated  maintainability 
OpCon,  with  roles, 
responsibilities,  workflows, 
logistics  management  plans  with 
budgets,  schedules,  resources, 
staffing,  infrastructure  and 
enabling  MPT  choices,  V&V  and 
review  procedures. 

Achieving  basic  People  CMM 
levels  2  and  3  maintainability 
practices  such  as  maintainability 
work  environment,  competency 
and  career  development,  and 
performance  management 
especially  in  such  key  areas  such 
as  V&V,  identification  &  reduction 

of  technical  debt. 

Advanced,  integrated,  tested  full- 

LC  Model-Based  Software  & 

Systems  (MBS&SE)  MPTs  and 
Maintainability-other-SQ 
tradespace  analysis  tools 
identified  for  use,  and  being 
individually  used  and  integrated. 

5 

Convergence,  involvement  of 
main  maintainability  success- 

critical  stakeholders.  Some 
maintainability  use  cases  defined. 
Rough  maintainability  OpCon, 
other  SCSHs,  staffing,  resource 
estimates.  Preparation  for  NDI 

and  outsource  selections. 

In  addition,  independent 
maintainability'  experts 
participate  in  project  evidence- 
based  decision  reviews,  identify 
potential  maintainability  conflicts 
with  other  SQs 

Advanced  full- lifecycle  (full-LC) 
O&M  MPTs  and  SW/SE  MPTs 
identified  for  use.  Basic  MPTs  for 
tradespace  analysis  among 
maintainability  &  other  SQs, 
including  TCO  being  used. 

4 

Artifacts  focused  on  missions. 

Primary  maintenance  options 
determined,  Early  involvement  of 
maintainability  SCSHs  in 
elaborating  and  evaluating 

maintenance  options. 

Critical  mass  of  maintainability 
SysEs  with  mission  SysE 
capability,  coverage  of  full  M-SysE 
skills  areas,  representation  of 
maintainability  success-critical- 

stakeholder  organizations. 

Advanced  O&M  MPT  capabilities 
identified  for  use:  Model-Based 
SW/SE,  TCO  analysis  support. 

Basic  O&M  MPT  capabilities  for 
modification,  repair  and  V&V: 

some  initial  use. 

3 

Elaboration  of  mission 

Operational  Concept  (OpCon), 
Architectural  views,  lifecycle  cost 
estimation.  Key  mission,  O&M, 

success-critical  stakeholders 

(SCSHs)  identified,  some 
maintainability'  options  explored. 

O&M  success-critical  stakeholders 
provide  critical  mass  of 
maintainability-capable  SysEs 

Identification  of  additional.  M- 

critical  success-critical 

stakeholders. 

Basic  O&M  MPT  capabilities 
identified  for  use,  particularly  for 
OpCon,  Arch,  and  Total  cost  of 
ownership  (TCO)  analysis:  some 

initial  use. 

2 

Mission  evolution  directions  and 

maintainability'  implications 
explored.  Some  mission  use  cases 
defined,  some  O&M  options 
explored. 

Highly  maintainability-capable 
Systems  Engineers  (SysEs) 
included  in  Early  SysE  team. 

Initial  exploration  of  O&M  MPT 
options 

Focus  on  mission  opportunities, 
needs.  Maintainability  not  yet 

considered. 

Awareness  of  needs  for  early 
expertise  for  maintainability, 
concurrent  engr'g,  O&M 
integration,  Life  Cycle  cost 

estimation 

Focus  on  O&M  MPT  options 

considered 
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1.2.2  Task  2:  Methods  and  Tools  Piloting  and  Refinement 

Other  universities  on  the  SQOTA  team  are  focusing  on  MMPTs  for  strengthening  the  SQ  aspects 
of  systems  engineering  in  the  context  of  the  ontology  and  recent  DoD  emphasis  areas  such  as 
Model-Based  Systems  Engineering  (MBSE),  Modular  Open  Systems  Architecture  (MOSA),  and 
Set-Based  Design  (SBD).  Below  is  a  summary  of  the  work  being  done  by  Gaty  Witus  at  Wayne 
State  University  with  TARDEC;  similar  work  is  being  done  by  Michael  Yukish  at  Pelm  State 
University  on  Naval  applications 

Wayne  State  U.  MMPTs.  We  are  working  with  the  US  Army  TARDEC  to  develop  tradespace 
models  supporting  key  ground  vehicle  system  acquisition  decisions.  We  are  leveraging  related 
work  in  a  separate  project  for  DARPA  on  the  "Ground  Vehicle  Experimental  Technologies"  (GVX- 
T)  program  (fig.  1).  The  expressed  need  was  for  tradespace  models  and  tools  to  make  armor 
protection  versus  mobility  tradeoffs  decisions.  Different  factors  enter  at  different  tradespace 
perspectives  (fig.  2).  The  chain  links  affordability  to  the  design,  capability,  and  mission 
tradespaces.  We  began  developing  a  quantitative  tradespace  model  for  ground  vehicles  by  first 
developing  a  qualitative  model  of  the  dependencies  alllong  design  decisions,  resulting  vehicle 
characteristics,  and  operational  capabilities  (fi2:  3). 


Figure  1  Model  of  combat  effectiveness  at  the  "tip  of  the  spear"  -  interaction  of  combat  mobility,  lethality,  and 
survivability  plus  the  impact  of  tactical  mobility  and  availability  of  reserve  on  combat  decision  and  outcomes 
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Strategic  &  Regional  Command 
Affordability:  How  many  Bn  can  we  afford  to  equip  and 
maintain  with  the  vehicle? 

Capability:  Does  it  fill  a  critical  gap  in  ability  to  conduct  2 
major  contingency  operations  simultaneously?  What  - 
CONOPS? 

Deployability:  Can  we  transport  a  BCT  to  theater  of 
operations  in  96  hrs"  A  Div  in  120  hrs? 

Endurance:  Can  BCT  operate  independently  for  72  hrs? 


Combat-  Co/PIt/Squad 


Combat  Mobility:  What  are  the  obstacle,  gap,  slope, 
soft-soil,  hill  climb,  water  crossing,  close  urban  terrain, 
cornering,  and  dash  capabilities? 

Lethality:  What  threat  vehicles  and  prepared  positions 
can  the  vehicle  destroy  (front;  flank)?  At  what 
elevation? 

Survivability:  What  threat  armaments  is  the  vehicle 
protected  against  (front;  flank;  rear)? _ 


Operational  Theater  Command 
Availability:  Whatfraction  of  vehicles  will  be  fully 
capable  during  the  theater  operation?  Transportability: 
How  long  does  it  take  to  redeploy  an  BCT  and  Bn  within 
theater?  Which  assets  can  transport  the  vehicle? 
Supportability  (operational  energy):  Fuel  consumption 
per  day  at  OPTEMPO? 

Effectiveness:  What  is  the  operational  effectiveness  of 
BCT  and  Bn? 


L 


Tactical  -  BCT  &  Bn' 

Combat  Effectiveness:  What  size  threat  force  can  the 
Co/Pit  /Squad  reliably  defeat  in  the  mission  and 
conditions  of  the  CONOPS? 

Tactical  Mobility:  How  long  does  it  take  to  redeploy  an 
Co/PIt/Squad  within  BCT  and  Bn  area?  What  is  the 
range  without  refueling? 

Endurance:  How  long  can  Co/Pit  /Squad  fight  at 
OPTEMPO  without  relief? 


Figure  2  Affordability-capability-design  tradespace  model  from  command  perspectives 


Characteristics 

Caliber  &  Elevation 
Stowed  Kills 
Dimensions  &  Aspect  Ratios 
Drive  Axel  Height 
Wheels  vs  Tracks 
learance,  Jounce  &  Rebound 
Ground  Pressure 
ower/Weight 
Weight 
Ballistic  &  Blast  Protection 


Capabilities 

■  Lethality 
Combat  Mobility 

actical  Mobility  l 
ransportability  1 
Supportability  I 
Availability  J 
Survivability 


Figure  3  Dependency  tradespace  model 

AFIT  and  NPS  MMPTs.  Another  SQOTA  collaborative  effort  in  the  MBSE  area  is  being  pursued  by 
David  Jacques  at  AFIT  and  Ray  Madachy  and  Kristen  Giammarco  at  NPS,  in  the  area  of  coordinated 
drones  performing  intelligence-surveillance-reconnaissance  (ISR)  functions.  Figure  4  provides  an 
example  of  their  modeling. 
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Figure  4  Example  AFIT-NPS  Coordinated-Drone  Operational  Model 

GTRI  MMPTs.  The  Georgia  Tech  Research  Institute  (GTRI)  has  also  successfully  transitioned  its 
SQOTA  (previously  called  ITAP,  for  ilities  Tradespace  and  Affordability  Project)  research  to  several 
DoD  organizations,  led  byTommer  Ender,  Valerie  Sitterle,  and  Daniel  Browne  at  GTRI.  Here  are 
two  examples. 

The  SQOTA  effort  seeks  to  better  analyze  the  tradespace  of  ilities  (e.g.,  affordability,  flexibility, 
reliability,  usability,  and  interoperability)  that  are  vitally  important  to  effective  Pre-Milestone  a 
tradespace  exploration  and  analysis  for  the  DoD.  Ilities,  now  more  commonly  referred  to  as 
system  qualities  (SQs),  are  frequently  under-emphasized  non-functional  properties  or  quality 
attributes  of  systems.  In  support  of  the  SQOTA  effort,  GTRI  has  investigated  new  methods  and 
constructs  for  design  exploration.  GTRI's  objectives  focus  on  methods,  processes,  and  tools  to 
support  analytical  formations  through  flexible  and  rationally  guided  workflows.  There  are  two 
primary  thrusts  to  the  research:  1)  methods  and  constructs  to  analytically  execute  formalisms, 
and  2)  processes  and  tools  that  help  operationalize  these  constructs  in  a  scalable  and  traceable 
manner. 
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GTRI's  guiding  vision  is  to  discover  a  "best"  set  of  options  through  a  synthesis  of  design  definition, 
tradespace  generation,  and  decision  analysis.  The  work  began  by  leveraging  previous  GTRI 
research  for  authoring  system  engineering  models  in  an  SysML-like  fashion  through  a  web 
browser  and  using  those  models  to  execute  design  tradespace  exploration.  For  SQOTA,  GTRI 
began  to  extend  these  capabilities  to  allow  feedbacks  within  the  design  process  and  compatibility 
with  NASA's  OpenMDAO  framework.  OpenMDAO  is  an  open-source  Multidisciplinary 
Design  Analysis  and  Optimization  (MDAO)  framework  developed  by  NASA  Glenn  and  Langley 
Research  Centers  for  use  as  an  integrated  analysis  and  design  enviromnent.  By  incorporating 
OpenMDAO  alongside  other  open  source  technologies,  GTRI  laid  the  groundwork  for  linking 
multiple  disparate  models  or  other  analysis  tools  in  a  single  design  structure  matrix.  As  this  work 
matured,  GTRI  focused  on  the  data  pipelining  aspects  of  how  to  combine  various  analytical  and 
decision  making  tools  into  a  comprehensive  but  flexible  framework.  By  allowing  decision  makers 
to  preemptively  sketch  and  organize  flexible  workflows  incorporating  these  elements  as  building 
blocks,  we  enable  a  more  effective  synthesis  of  design  definition,  tradespace  generation,  and 
decision  analysis. 

Alongside  these  efforts,  GTRI  collaborated  with  ITAP  team  members  at  the  University  of  Southern 
California  and  Naval  Postgraduate  School  to  interface  existing  cost  models  with  SysML-based 
systems  models.  Unifying  these  areas  of  work  provides  the  basis  for  next-  generation  cost 
modeling  approaches  that  offer  estimates  of  the  system  engineering  effort  costs,  measure  the 
complexity  of  the  effort,  and  show  how  those  aspects  relate  to  one  another.  This  work  is  being 
extended  to  system  of  systems  problems  with  building  blocks  in  SysML  that  leverage  the 
COSYSMO-SoS/COCOMO  legacy,  and  being  validated  against  healthcare  SoS  case  studies. 

Throughout  the  development  process  and  method  inclusion,  this  effort  and  its  future  maturation 
seeks  to  preserve  an  open  framework  and  approach  that  promotes  quantitative  and  qualitative 
transparency  of  the  tradespace  refinement.  Our  goals  are  to  ensure  that  the  workflow  and 
toolset  support  easy  inclusion  of  analytical  constructs  that  may  be  developed  on  other  ITAP 
efforts  to  evaluate  different  ilities  in  different  ways.  Our  work  towards  unifying  the  design, 
decision  variable,  and  objective  spaces  in  a  flexible  framework  is  being  leveraged  and  matured 
under  the  DoD's  Engineering  Resilient  Systems  program  to  support  effective  and  efficient  design 
and  development  of  complex  engineered  systems  across  their  lifecycle. 
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Scenario  Based  Needs  Context 


Flexible  workflows,  integrating  MBSE  and  MDAO 


Figure  5  Transitioning  ITAP  -  Successfully  transitioned  methods  and  approaches  developed  under  ITAP  to  other 

DoD  programs 

Engineered  Resilient  Systems  (ERS)  is  one  of  seventeen  DoD  Communities  of  Interest  led  by  the 
US  Army  Corps  of  Engineers.  One  of  its  primary  goals  is  to  develop  an  integrated,  trusted, 
computational  environment  supporting  all  phases  of  the  DoD's  acquisition  and  operational 
analysis.  This  will  result  in  a  series  of  government  owned  and  hosted  tools  to  support.  One  of 
these  tools,  the  "ERS  TradeBuilder"  is  built  by  GTRI  to  conduct  executable,  model  based 
systems  engineering  and  support  trade  studies. 

The  "Define"  segment  of  the  ERS  TradeBuilder  enables  a  user  to  use  Model  Based  Systems 
Engineering  best  practices.  Users  can  define  the  Problem,  to  include  decomposing  system 
requirements  from  operational  needs.  Users  can  further  define  the  System  of  interest,  to 
include  a  physical  decomposition  according  the  MIL-STD-881C  Work  Breakdown  Structure. 
Finally,  users  can  define  the  Analysis  technique  which  will  be  used  (to  include  Modeling  and 
Simulation). 

Next ,  the  "Execution  "portion  of  the  toolset  enables  a  user  to  run  a  tradespace  using  high 
fidelity  Modeling  and  Simulation  leveraging  the  DoD's  High  Performance  Computing  (HPC) 
assets.  Users  can  define  ranges  for  the  attributes  of  interest  and  select  the  appropriate  HPC 
asset  to  leverage. 

The  final  component  of  the  ERS  TradeBuilder  is  "Explore",  where  a  user  can  conduct  a  wide 
variety  of  analysis  to  include  Analysis  of  Alternatives,  requirements  feasibility,  and  other  trade 
studies.  The  Explore  portion  of  ERS  TradeBuilder  enables  a  user  to  customize  a  dashboard  with 
a  number  of  analytical  tools.  ERS  TradeBuilder's  Exploration  capability  enables  a  user  to 
customize  a  dashboard  with  relevant  analytical  tools  such  as  interactive  scatter  plots, 
histograms,  and  stop-light  scoring  charts.  The  user  can  further  score  the  candidate  systems 
according  to  overall  Value  using  utility  functions. 
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ENGINEERED  RESILIENT  SYSTEMS 


DEPARTMENT  OF  DEFENSE 

TradeBuilder 


Figure  6  Engineered  Resilient  Systems  "TradeBuilder" 


1.2.3  Task  3:  Next-Generation,  Full-Coverage  Cost  Models  (USC,  NPS) 

Trends  affecting  system  and  software  engineering  practices  such  as  internets  of  things,  3D 
printing,  cloud  services,  big-data  analytics,  autonomic  and  learning  systems,  agile  methods, 
and  asymmetric  threats,  such  as  for  cyber  security,  present  challenges  for  DoD  systems 
and  software  engineering  practices,  and  also  for  estimating  their  costs  and  their  impact 
on  Affordability.  The  SQOTA  Next-Generation  Cost  Models  effort,  with  Co-PIs  Barry  Boehm 
and  JoAnn  Lane  at  USC  and  Ray  Madachy  at  NPS,  has  made  significant  progress  in  defining 
next-generation  versions  of  the  COCOMO  II  software  cost  model  (COCOMO  III,  led  by  Brad 
Clark  at  USC),  and  the  COSYSMO  2.0  systems  engineering  cost  model  (COSYSMO  3.0,  led 
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by  Jim  Alstad  at  USC).  For  COCOMO  III,  we  have  concluded  that  there  will  be  no  single 
model  that  is  good  for  estimating  all  of  the  challenge  areas  above,  and  are  prioritizing  to 
create  an  initial  version  that  best  fits  most  of  DoD's  major  project  types. 

For  COSYSMO  3.0,  in  2016  we  achieved  a  major  milestone  in  completing  an  Expert-Based 
Model.  Our  workshops  and  weekly  expert  telecons  with  industry,  government,  and 
academia  representatives  resulted  in  gathering  complete  set  of  expert  judgments  about  the 
form  and  parameters  of  the  model,  so  that  we  now  have  a  fully  developed  interim  model. 
The  remaining  step  is  to  gather  actual  project  data  and  combine  that  with  the  expert 
opinions  to  yield  the  final  model.  The  2016  events  at  which  we  had  half-day  working  group 
sessions  for  COSYSMO  3.0  and  often  COCOMO  III  were  the  Army-  and  Navy-sponsored 
Practical  Systems  Measurement  User  Group  (February),  the  Ground  Systems  Architecture 
Workshop  (March),  USC  CSSE's  Annual  Research  Review  (March),'the  Navy  and  NGA- 
sponsored  Software  and  IT  Cost  Analysis  Solutions  Team  meeting  (August),  and  USC  CSSE's 
COCOMO  Forum  (October). 

The  net  result  is  shown  in  Figure  7,  which  shows  the  expert-consensus-determined  values 
of  the  COSYSMO  3.0  effort  multiplier  ratios  (EMRs)  of  the  revised  cost  drivers:  the  ratio  of 
their  maximum  value  to  their  minimum  value.  Generally,  cost  drivers  with  larger  EMRs  are 
more  important  to  a  cost  estimate  than  cost  drivers  with  smaller  EMRs.  DFR  is  Development 
for  Reuse. 


Cost  Driver  Impacts  (EMRs)  in  COSYSMO  3.0  v35 
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Figure  7  Expert-Based  COSYSMO  3.0  Effort  Multiplier  Ratiosl.3  Future  Plan  for  Phase  6 
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Phase  6  will  continue  the  three  primary  tasks  (Tasks  1-  3).  Overall  summaries  of  the  tasks  are 
provided  next,  followed  by  summarized  Phase  6  deliverables. 

Task  1.  Research  and  Develop  SQ  Scientific  Foundations 

This  task  will  continue  to  expand,  coordinate,  and  evolve  two  complementary  ontologies  for  the 
SQs:  a  Product  Quality,  means-ends  framework  being  evolved  at  USC,  and  a  Quality-in-Use 
semantic  framework  for  change-oriented  SQs  being  evolved  at  MIT,  with  formal  definitions  for 
each  being  evolved  at  U.  Virginia.  Each  will  have  compatible  definitions  of  the  key  DoD  SQs,  and 
associated  methods  and  tools  for  making  formalizations  accessible  to  the  systems  engineering 
research  and  practitioner  communities  for  validation  against  the  needs  of  practice.  The  Product 
Quality  framework  will  continue  to  populate  its  synergy  and  conflict  relationships  among  the  SQs; 
to  expand  the  quantification  of  the  synergies  and  conflicts;  and  to  refine  the  prototype  tools  for 
representing  and  applying  the  results.  It  will  also  develop  complementary  views  for  addressing 
DOD  high-priority  SQ-related  issues  dealing  with  uncertainties  such  as  sources  of  change,  early 
cost-effectiveness  analysis,  and  Total  Cost  of  Ownership  analysis.  It  will  also  develop  and  apply 
scientific  theories  to  validate  the  capabilities  and  consistency  of  the  frameworks,  models, 
methods,  processes,  and  tools  researched  and  developed. 

Research  team:  Primarily  USC  (lead),  MIT UVA 

Task  2.  SQ  Methods,  Processes,  and  Tools  (MPTs)  Piloting  and  Refinement 


This  task  will  follow  up  on  the  engagements  with  DoD  organizations  pursued  in  Phases  2  through 
4,  to  continue  to  pilot  the  application  of  SERC  methods  and  tools  to  DoD-system  SQ  tradespace 
and  affordability  issues,  particularly  in  the  cyber-physical-human  systems  and  economic  analysis 
areas.  The  methods  and  tools  will  continue  to  be  refined,  based  on  the  results  of  the  pilot 
applications.  Representative  activities  will  include  efforts  to: 

•  Experiment  with  tailoring  existing  or  new  SQ  tradespace  and  affordability  MPTs  for  use 
by  early  adopter  organizations 

•  Train  early  adopters  in  their  use,  monitor  their  pilot  usage,  and  determine  areas  of 
strengths  and  needed  improvements,  especially  in  the  MPTs'  ilities 

•  Extend  the  MPTs  to  address  the  top-priority  needed  improvements 

•  Work  with  early  adopters  to  help  transition  the  improved  MPTs  into  their  use 

•  Identify  and  pursue  further  improvements  for  the  early  adopters  or  for  more  general 
usage 

The  Primary  Research  Team  will  be  as  follows:  Wayne  State  U.  {lead),  AFIT,  GT,  NPS,  PSU,  and 
USC.  MIT  and  UVA  will  selectively  participate  based  on  their  Foundations  MPTs. 

•  Army  TARDEC,  Navy  NAVSEA,  NAVAIR,  SPAWAR:  WSU,  GT,  NPS,  PSU,  USC 

•  USMC:  GaTech,  others 

•  USAF:  AFIT-ASC;  USC-SMC 
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Task  3.  Next-Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles. 


Beginning  with  work  in  the  space  domain  with  USAF/SMC  and  the  Aerospace  Corp.,  this  task  has 
refocused  on  the  two  areas  for  which  the  needs  and  availability  of  data  are  the  strongest:  overall- 
DoD  next-generation  versions  of  the  COCOMO  model  for  software  cost  and  schedule  estimation 
and  the  COSYSMO  cost  model  for  systems  engineering  cost  and  early-phase  schedule  estimation. 


Research  team:  Primarily  USC  (Lead),  NPS,  AFIT 


Task  Statements  for  Phase  6 


Task  1.  System  Quality  (SQ)  Foundations 

MIT.  Continue  work  with  Mitre  on  extension  and  application  of  Changeability  semantic  basis. 
Extend  semantic  basis  to  selected  Mission  Effectiveness  qualities.  Develop  capabilities  for  using 
semantic  basis  in  Epoch-Era  analysis. 

USC.  Extend  Maintainability  data  analytics,  using  cloud  services  to  track  patterns  of  increase  and 
decrease  of  Mean  Time  to  Repair  and  Technical  Debt.  Continue  experimental  comparison  of 
automated  and  human  Maintainability  assessment  approaches,  and  develop  guidelines  for  their 
complementary  use. 

Pilot,  evaluate,  and  refine  SYSte m/Software  Maintainability  Readiness  Framework  (SMRF)  on 
representative  projects. 

Explore  similar  elaboration  of  data  analytics  and  Synergy  &  Conflict  analysis  for  Dependability 
attributes:  Reliability,  Availability,  Safety,  Security,  Robustness,  Graceful  Degradation.  Use 
results  to  participate  in  update  of  SEBoK  Specialty  Engineering  update. 

Explore  variant  definitions  of  Resilience  as  combinations  of  Dependability  and  Changeability 
variants. 

USC-UVa.  Continue  elaboration,  exploratory  application,  evaluation,  and  refinement  of  formal 
System  Quality  definitions. 

Task  2.  System  Quality  {SQ)  Methods,  Processes,  and  Tools  (MPTs)  Piloting  and  Refinement 
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AFIT,  NPS.  Continue  to  elaborate  DoDAF  representation  of  partially-executable  Operational  and 
Systems  Architecture,  using  associated  Model-Based  Systems  Engineering  (MBSE)  tools  such  as 
SysML.  Monterey  Phoenix  (MP),  Cooperative  unmanned  Surveillance  System  (CUSS),  COSYSMO, 
and  further  capabilities  such  as  set-based  modeling  and  Orthogonal  Defect  Classification 
COQUALMO  for  automated  analysis  of  off-nominal  UAV  ISR  operational  scenarios,  such  as 
jamming,  evasion,  and  camouflage.  Apply,  evaluate,  refine,  and  extend  system  and  models  based 
on  experimentation  using  nominal  and  off-nominal  UAV  ISR  operational  scenarios  at  Air  Force 
drone-based  mission  testbed. 

Examples  of  extensions  from  early  Phase  5  studies  include  going  from  a  basic  multi-vehicle 
architecture  using  Small  UAS  to  locate,  confirm,  track  and  engage  widely  dispersed  targets.  Basic 
architectural  variations  included  numbers  of  vehicles,  quality  (performance  measures)  of  the 
sensors,  and  C2  variations  that  considered  operator-in-the-loop  versus  full  autonomous 
operation.  For  Phase  6,  we  are  developing  a  more  complex  architecture  based  on  Small  UAS 
providing  remote  targeting  support  for  larger,  standoff  vehicles.  Architectural  views  included 
requirements  diagrams,  functional  decomposition  (hierarchical),  activity  diagrams,  block 
definition  diagrams,  and  interface  definition  both  across  system  elements  and  at  the  subsystem 
level  within  a  system  element.  This  level  of  definition  will  include  operational  threads, 
requirements,  and  interfaces  at  the  appropriate  decomposition  level  as  direct  input  to 
parametric  cost  models.  The  thorough  and  detailed  SysML  model  comprising  size  inputs  will 
enable  a  COSYSMO  cost  estimate  and  extrapolated  full  lifecycle  cost,  amenable  to  later 
architectural  variations. 

GTRI.  For  Phase  6,  GTRI  will  extend  and  mature  the  proof-of-concept  PAW  framework  and 
associated  tools  developed  in  Phase  5.  Specifically,  we  will  investigate  different  approaches  to 
capturing  and  characterizing  dimensions  of  operational  context  relevant  to  DoD  materiel  design 
and  development.  This  includes  methods  and  processes  through  which  the  PAW  framework  may 
help  us  more  efficiently  and  effectively  (a)  generate  contextual  tradespace  data  driven  by  both 
the  physics  of  the  problem  as  well  as  the  operational  needs,  and  (b)  address  contextual  and  other 
non-simple  sources  of  uncertainty  in  trades  pace  analysis.  Together,  these  thrusts  will  help  the 
DoD  community  address  more  complex  environments,  operational  scenarios,  and  multi-  criteria 
analyses  within  an  executable  toolset.  To  accomplish  these  goals,  GTRI  will  focus  on  leveraging 
the  flexibility  and  customizable  modularity  of  the  PAW  framework  to  investigate  these  methods 
and  processes  by  comparing  and  contrasting  SE  approaches  across  problem  scales.  We  will 
compare  these  methods  and  processes  to  the  traditional,  linear  tradespace  generation  and  multi¬ 
criteria  decision  analysis  techniques.  The  goal  is  to  address  how  we  can  intelligently  create  the 
data  we  need  to  answer  SE  questions  critical  to  understanding  what  is  driving  decisions  for  DoD 
decision  makers.  These  efforts  will  be  piloted  using  a  Jupyter  notebook  platform  to  promote 
collaborative  use  and  pathways  to  interoperability  for  the  SERC  SQOTA  team  members  and  DoD 
sponsors.  We  will  demonstrate  application  of  these  capabilities  to  sample  problems  illustrative 
of  the  DoD  challenges. 
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WSU.  WSU  will  work  with  TARDEC  to  develop  a  tradespace  analysis  model  for  ground  vehicle 
development  in  the  context  of  the  Third  Offset  Strategy  and  the  US  Army  Combat  Vehicle 
Modernization  Strategy. 

WSU  Task  1:  Tradespace  Framework  for  Collaborative  Concept  Development  and  Design 

This  task  will  be  conducted  in  collaboration  with  US  Army,  TARDEC,  working  with  PhD  candidates 
with  SMART  scholarships  from  TARDEC. 

Task  la.  We  will  expand  the  stakeholder-perspective  tradespace  model  previously  developed  to 
include  the  other  factors  and  relationships  described  in  the  Army  Combat  Vehicle  Modernization 
Strategy. 

Task  lb.  We  will  refine  the  statements  of  the  stakeholder  perspectives  to  address  impact  on  and 
contribution  to  combined  arms  operations  considerations  since  no  vehicle  is  designed  to  or 
operates  in  isolation,  but  as  part  of  a  combined  arms  "System  of  Systems." 

Task  lc.  We  will  convert  the  dependency  diagram  format  to  relational  statements  in  executable 
code  to  trace  logical  dependencies,  and  to  evaluate  quantitative  dependencies  (when 
quantitative  dependency  relationships  are  feasible  and  available). 

Task  Id.  We  will  complete  expanding  the  model  with  a  set-based  design  value  function  in  order 
to  address  adaptability  of  modular  systems,  and  the  value  of  the  ability  to  reconfigure  and 
repurpose  a  vehicle  platform  as  need  to  address  potential  future  adversary  conditions  and 
adaptations.  This  work  will  complete  the  progress  towards  formulating  a  set-based  value  function 
that  treats  the  vehicle  as  developed  and  procured  as  defining  a  set  of  potential  future 
configurations,  mission  module  and  kit  upgrades,  to  deal  with  uncertain  and  adaptive  adversaries 
over  the  useful  life  of  the  system.  A  graduate  student  is  completing  his  PhD  dissertation  on  this 
topic,  and  will  defend  in  2017. 

The  set  value  function  and  the  stakeholder-perspective  model  are  the  key  elements  to 
envisioning  collaborative  "set-based"  design  in  ground  vehicle  conceptualization  and 
specification. 


WSU  Task  2:  Quantitative  Tradeoff  Models 

This  task  will  be  restricted  in  scope  to  mobility-survivability-lethality  tradespace  from  the  tactical 
and  combat  perspectives.  The  task  will  build  on  and  extend  the  mobility-survivability  tradespace 
state-transition  computational  modeling  conducted  for  DARPA  as  part  of  the  GVS-T  project.  It 
will  extend  the  computational  model  to  include  lethality,  as  shown  in  Figure  3. 
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This  is  the  minimal  viewpoint  from  which  to  address  tradeoffs  between  combat  mobility  and 
tactical  mobility.  This  has  been  an  historic  gap.  Existing  methods,  tools  and  procedures  to  develop 
mobility  performance  specifications  (P-Specs)  have  not  addressed  the  interaction  effects  and 
dependencies.  The  combat  perspective  is  needed  to  justify  combat  mobility  characteristics,  while 
the  tactical  perspective  of  reinforcement  and  exploitation  is  needed  to  justify  tactical  mobility. 
The  extension  to  quantifying  the  tradespace  adds  (a)  lethality  and  (b)  reinforcement  to  the 
quantitative  model  developed  for  the  DARPA/TARDEC  GVX-T  project. 

The  two  different  mobility  activities  are  needed  to  address  wheel-vs-track  issues.  Armor 
protection  and  combat  mobility  have  a  complex  interaction.  Armor  adds  weight  which  decreases 
mobility  which  increases  the  time  exposed  to  threats  and  the  time  to  circumvent  threat 
obstacles.  Mobility  enables  evasive  maneuver  which  improves  survivability  even  when  the  threat 
weapon  overmatches  the  US  armor. 

Armor  is  effective  only  when  it  overmatches  the  threat  weapon-  which  leads  to  step  functions  in 
the  value  of  armor.  The  distribution  of  armor  is  also  part  of  the  tradespace  (frontal  arc  vs  360 
protection)  -  mixed  solutions  are  possible.  Combat  mobility  impacts  the  time  US  vehicles  have 
frontal  vs  side  exposure.  Lethality,  enabled  by  combat  mobility,  is  also  part  of  the  equation. 
Historical  efforts  to  develop  a  "light  tank"-  e.g.,  the  Sheridan,  light  enough  for  air  transport  with 
lethal  capability,  but  lightly  armored-  proved  ineffective  and/or  inefficient.  The  MPF  is  at  risk  of 
the  same  fate.  The  land  combat  community  has  not  resolved  tradeoffs  between  operational  and 
tactical  mobility  vs  combat  mobility  in  the  context  of  the  armor  and  armament  tradeoffs,  and 
threats  &  operating  conditions. 

The  Third  Offset  vision  of  operating  challenges  includes  operations  in  dense  urban  environments 
and  megacities  in  which  only  smaller,  lighter  vehicles  can  operate  without  air  superiority  or  air 
support.  We  cannot  solve  this  tradespace  decision  challenge  in  the  next  12  months  under  limited 
budget.  Deployability  and  in-theater  transportability  are  overriding  concerns  from  the  strategic 
perspective,  as  are  (a)  the  time  and  cost  to  Initial  Operational  Capability,  and  (b)  the  time  and 
cost  to  transition  the  force.  These  are  complex  issues,  beyond  the  budget  of  this  task. 

Within  the  budget  of  this  task,  we  propose  to  deliver  a  quantitative  trades  pace  model  for 
propulsion,  armor  and  armament  in  the  context  of  combat  and  tactical  operations.  This  will  be 
demonstrated  in  the  context  of  armor  and  mobility  tradeoffs. 


WSU  Task  3:  Autonomy  and  Manned-Unmanned  Teaming  to  Enlarge  the  Tradespace 
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Enlarging  the  tradespace  happens  when  technologies  enable  possibilities  that  were  previously 
infeasible.  Given  the  resource  constraints  we  can  only  begin  to  scratch  the  surface  of  the 
tradeoffs  that  autonomy  and  manned-unmanned  teaming  enables. 

In  this  task  we  will  use  the  models  and  frameworks  produced  from  tasks  1&  2  as  the  model  to 
explore  the  impact  of  removing  or  reducing  the  crew  on  cost  and  operational  effectiveness. 
Within  this  limited  scope,  we  will  continue  to  focus  on  mobility,  lethality  and  survivability  at  the 
system  design  level,  and  at  higher  levels  of  strategic,  operational  and  tactical  mobility. 

The  critical  tradespace  questions  are  (a)  what  "autonomous"  and  "manned-unmanned  teaming" 
are  needed  for  what  operations?,  and  (b)  how  does  "autonomous"  and  "manned-  unmanned 
teaming"  expand  the  design  space  and  change  the  tradespace? 

These  questions  are  critical  for  the  ground  vehicle  development  community  to  respond  to  the 
Third  Offset  Strategy  to  employ  autonomy  technologies  to  achieve  decisive  advantage. 


PSU.  Continue  to  exercise,  evaluate,  refine,  and  extend  set-based  design  (SBD)-enhanced  tools 
for  ground-based,  sea-based,  and  air-based  vehicles  and  their  mission  performance,  in  concert 
with  NAVSEA  and  NAVAIR,  including  extensions  for  a  tradespace  decision  structure  that  identifies 
key  tradeoff  analyses  and  the  system  development  decisions  that  they  support. 

All.  Identification  and  initial  implementation  of  interoperability  facilitation  capabilities  among 
SQMPTs. 

Task  3.  Next-Generation  Cost  Estimation  Models 


USC,  NPS. 


COSYSMO  3.0.  Next-Generation  Constructive  Systems  Engineering  Cost  Model. 

Phase  5  completed  the  definition  of  COSYSMO  3.0,  including  experience-based  refinement  of  the 
basic  model  and  extensions  for  the  effects  of  system  reusability,  reuse,  requirements  volatility, 
and  systems  of  systems  integration,  and  developing  an  initial  quantitative  model  based  on 
expert-  consensus  definitions  of  the  quantitative  sizing,  cost  driver,  and  scale  factor  parameters. 
Phase  6  will  evaluate  basic  usage  of  the  initial  model,  and  gather  data  on  completed  projects  for 
calibration  to  project  data  and  Bayesian  integration  of  the  expert-based  and  data-based  model 
parameters.  Activities  will  include  experience-based  refinement  workshops  at  the  18th  Army- 
Navy-CARD  Practical  Systems  Measurement  Users  Group  (PSMUG)  Workshop  in  June  2017,  the 
32"d  USC  COCOMO/System  and  Software  Cost  modeling  (COCOMO/S&SCM)  Forum  in  October 
2017,  and  the  21'1  Aerospace  Corp.-USAF  Ground  Systems  Architecture  Workshop  (GSAW)  in 
February-March  2018. 
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COCOMO  III.  Next-Generation  Constructive  Software  Cost  Model.  Phase  S  determined  that  a 
single  software  cost  model  would  not  be  able  to  cover  the  wide  variations  among  software 
development  for  major  systems,  domain  cloud-services  applications,  mobile  web-based 
applications,  and  various  forms  of  agile.  Kanban,  SAFE,  and  DevOps  projects.  For  Phase  6,  the 
focus  will  be  on  a  version  of  COCOMO  III  for  major  systems,  including  expert-based  parameter 
definitions,  an  initial  expert-based  quantitative  model,  and  similar  workshops  at  the  PSMUG, 
COCOMO/S&SCM,  and  GSAW  events. 

USC,  NPS,  GTRI.  GTRI  and  collaborators  USC  and  NPS  outlined  a  SysML  cost  modeling  roadmap  in 
the  201S-2016  report.  For  Phase  6,  the  proposed  next  steps  for  2017-2018  along  that  roadmap 
include  maturing  this  work  along  the  lines  of  capability  extensions  using  new  tools,  infusing 
context  into  those  capabilities,  and  demonstrating  the  capabilities  in  case  studies  relevant  to  the 
DoD.  For  the  extension  work,  GTRI  proposes  to  create  the  next  round  of  front-  ends  based  on 
OpenMBEE  and  add  new  functionality  to  the  SysML  cost  modeling  building  block  library. 
Specifically,  the  work  will  focus  on  creating  new  building  blocks  to  support  (a)  software  cost 
modeling  (via  the  COCOMO  technique),  (b)  hardware  cost  modeling  (via  the  Advanced  Missions 
Cost  Model  (AMCM)  technique),  and  (c)  adding  COSYSMO  2. 0/3.0  aspects  including 
considerations  for  reuse,  risk,  etc.  The  capabilities  will  focus  on  addressing  the  broader 
affordability  tradespace  context  and  the  feasibility  of  auto-generating  inputs  to  COSYSMO  etc. 
from  a  regular  SysML-  based  system  model.  GTRI  will  work  with  USC  and  NPS  to  demonstrate 
application  of  these  capabilities  to  case  studies  relevant  to  the  sponsor  and  investigate  having 
the  growing  community  of  OpenMBEE  users  as  beta  testers  of  the  SysML  cost  modeling  building 
blocks. 


Design  and  Demonstrate  ISR  UAV  Tradespace.  The  power  of  new  tools  such  as 
MagicDraw  and  ModeiCenter  is  the  integration  with  analysis  tools,  across  a  network 
or  on  the  same  machine.  SySML  has  traditionally  only  captured  and  documented  the 
operational  concept,  system  requirements,  activities/tasks,  organizations  and 
information  flows,  including  possible  physical  instantiations.  New  MBSE  tools 
facilitate  analysis  such  as  optimization,  simulation,  design  of  experiments, 
assessment,  sensitivity  analysis  and  statistical  hypothesis  testing  and  regression. 

For  this  project,  such  trades  and  characterizations  could  examine  collaborative  and 
vehicle  swarming  algorithms,  increasing  autonomy  on  multi-  vehicle,  and  single 
operator  operations.  Likewise,  the  effects  on  environmental  variables  within  the 
architecture,  such  as  communications  and/or  GPS  jamming,  air  defenses,  evasion, 
camouflage,  and  other  factors  could  define  the  scenarios.  These  types  of  trades 
demonstrate  how  a  business  case  for  varying  technologies,  capabilities  or  designs 
could  be  accomplished,  if  cost  information  is  included. 

Develop  life  cycle  cost  model  interfaces  for  the  various  components  of  the 
architecture,  and  embed  them  within  a  larger  stochastic  life  cost  estimating 
approach  to  evaluate  cost  effectiveness  in  an  uncertain  future  environment. 

Cost  data  will  be  attached  to  every  actor  and  event  in  every  possible  scenario, 
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computing  the  cost  of  each  scenario,  were  it  to  occur.  A  probability  of  occurrence 
for  each  possible  scenario  will  be  generated,  providing  for  highly  refined  overall 
cost  estimates  (for  operations,  for  maintenance,  and  perhaps  earlier  lifecycle 
phases)  within  a  specified  confidence  interval. 

With  the  above  steps,  this  application  will  use  MP  with  cost  modeling  to  enhance  tradespace 
analysis  of  UAV  systems.  The  executable  integrated  architecture  will  provide  for  evaluation  of 
UAV  technologies  and/or  design  alternatives  across  a  range  of  operational  scenarios  utilizing 
MBSE  tools  and  notations. 

Current  commercial  MBSE  tools  support  creation  of  a  very  small  number  of  operational  scenario 
instances  compared  to  what  is  possible.  The  validity  of  these  tradespace  analyses,  however, 
stands  to  be  substantially  improved  by  expanding  the  number  of  scenario  variants  considered,  to 
include  a  wider  range  of  possible  nominal  and  off  nominal  behaviors  in  both  the  system  under 
design  and  the  environment.  In  particular,  resulting  cost  estimates  (e.g.,  for  UAV  software 
development,  as  well  as  UAV  missions  during  operations  &  maintenance)  are  impacted  by 
underrepresentation  of  possible  scenarios  and  the  lack  of  probability  data  on  those  scenarios. 
One  of  our  objectives  is  to  increase  the  resolution  of  source  data  used  for  cost  model 
computations.  We  will  compare,  contrast,  and  possibly  integrate  methods  in  Phase  5  for  cost 
modeling  in  MP  and  SysML  depending  on  the  Phase  4  results. 

The  technical  approach  involves  cycling  AFIT-developed  operational  scenarios  through  the  MP 
modeling  process,  whereby  alternate  events  are  captured  for  each  actor  in  each  scenario.  This 
will  produce  a  superset  of  scenario  variants  from  the  behavior  models,  suitable  for  input  to 
tradespace  analysis  models  and  cost  model  hooks  developed  in  SysML. 

With  this  we  can  capture  lifecycle  cost  attributes  for  each  function  point  in  the  architecture, 
based  on  internal  and  external  interactions  in  the  MP  models.  We  can  also  capture  cost  attributes 
for  each  actor  and  each  event  in  each  generated  scenario  for  use  in  mission  cost  effectiveness 
analyses.  Phase  6  will  develop  further  improvements  to  MP.  Based  on  Phase  4  results  these  may 
include  an  improved  event  trace  generator  and  a  user-friendly  GUI. 

Task  3.  Next-Generation,  Full-Coverage  Cost  Estimation  Model  Ensembles 


NPS  will  continue  extending  the  scope  and  tradespace  interoperability  of  cost  models  and  tools 
in  Phase  6.  This  is  based  on  stakeholder  feedback  in  earlier  phases  for  parametric  model 
enhancements  and  tool  automation  improvements. 

Leveraging  Phase  5  cost  driver  research.  Phase  6  will  develop  prototype  systems  engineering  and 
software  cost  models  and  tools  for  piloting  and  refinement,  and  extend  the  estimation 
capabilities  toward  full-coverage  in  conjunction  with  USC. 
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The  cost  modeling  activities  will  engage  domain  experts  for  Delphi  estimates,  evolve  baseline 
detailed  definitions  of  the  cost  driver  parameters  and  rating  scales  for  use  in  data  collection,  and 
gather  initial  data  and  determine  areas  needing  further  research  to  account  for  wide  differences 
between  estimated  and  actual  costs.  Phase  6  will  continue  the  extension  in  scope  of  the  models 
and  tools  and  their  piloting  and  refinement. 

For  tool  interoperability  we  will  integrate  cost  models  in  different  ways  with  MBSE  architectural 
modeling  approaches  and  as  web  services  (also  part  of  Task  2  piloting).  We  will  also  automate 
systems  and  software  risk  advisors  that  operate  in  conjunction  with  the  cost  models. 

We  will  expand  on  earlier  phase  results  for  cost  modeling  web  services.  We  previously  developed 
a  working  prototype  web  service  for  Orthogonal  Defect  Classification  Constructive  Quality  Model 
(ODC  COQUALMO)  supporting  tool  interoperability  (costing  in  the  cloud).  COQUALMO  was 
demonstrated  in  Phase  lwith  only  a  subset  of  cost  factors.  Per  interested  stakeholders  we  will 
develop  full  implementations  of  selected  parametric  cost  models  in  Phase  6. 

NPS  will  provide  domain  expertise  to  USC  and  Georgia  Tech  for  the  SysML  cost  model  integration 
effort.  We  will  add  the  COCOMO  software  cost  model  formulas,  and  risk  assessment  capabilities 
for  Expert  COSYSMO  Error!  Reference  source  not  found,  and  Expert  COCOMO  Error!  Reference 
source  not  found...  In  Phase  5,  and  we'll  continue  those  and  evaluate  Monte-Carlo  approaches 
within  SySML. 

This  task  is  also  tied  to  Task  2  piloting  with  MBSE  cost  model  interfaces.  In  Phase  6  we  will  assess 
MP  for  automatically  providing  cost  information  from  the  architectural  models.  This  is  analogous 
to  the  SySML  method  for  extracting  attributes  and  they  will  be  compared.  MP  will  be  used  to 
extract  software  sizing  information.  It  will  generate  function  point  measures  which  will  be  input 
into  COCOMO.  We  will  also  assess  how  MP  architectural  elements  can  be  mapped  into  systems 
engineering  cost  model  inputs. 


Appendix  A:  List  of  Publications  Resulted 


1-Books 

B.  Boehm,  J.  Lane,  S.  Koolmanojwong,  R.  Turner,  "The  incremental  Commitment  Spiral  Model," 
Chinese  language  version. 

2  -Papers  in  peer-reviewed  journals 

Accepted  to  INCOSE  Systems  Engineering  Journal  Special  Issue  of  best  papers  from  CSER  2016: 
B. Boehm,  C.Chen,  L.Shi,  K.Srisopha,  "Maintainability  Readiness  Levels  for  Software-Intensive 
Systems,"  Date  TBD,  2017. 
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-  Papers  in  conferences  with  full  paper  review 


R.  Alfayez,  C.  Chen,  P.Behnamgader,  K.  Srisopha,  B.  Boehm,  "An  Empirical  Study  of  Technical 
Debt  in  Open-Source  Software  Systems,  Proceedings,  CSER  2017. 

B.  Boehm,  "Improving  and  Balancing  Software  Qualities,"  Technical  Briefing,  ICSE  2016.  B. Boehm, 

C. Chen,  L.Shi,  K. Srisopha,  "Maintainability  Readiness  Levels  for  Software-Intensive 
Systems,"  CSER  2016,  March  2016. 

B. Boehm,  C.Chen,  L.Shi,  K. Srisopha,  "The  Key  Roles  of  Maintainabili  ty  in  an  Ontology  for  System 
Qualities,"  INCOSE  International  Symposium,  July  2016. 

Celia  Chen,  Reem  Alfayez,  Kamonphop  Srisopha,  Barry  Boehm  and  Lin  Shi,  "Why  is  it  Important  to 
Measure  Maintainability,  and  what  are  the  Best  Ways  to  Do  it?,"  accepted  poster  paper,  ICSE  2017. 

B.  Boehm,  "Software  Cost  Estimation  Meets  Software  Diversity,"  accepted  as  Technical  Briefing,  ICSE 
2017. 

Chen,  C.,  Alfayez,  R.,  Srisopha,  K.,  Shi,  L.,  and  Boehm,  B.  "Evaluating  Human-Assessed  Software 
Maintainability  Metrics."  In  Proceedings,  NASAC  2016. 

Shi,  L.,  Chen,  C.,  Wang,  Q.,  &  Boehm,  B.  "Is  It  a  New  Feature  or  Simply  "Don't  know  yet"?  On  Au 
tomated  Redundant  OSS  Feature  Requests  Identification."  In  Proceedings,  ACM  RE  Next  2016. 

Wu,  B.,  Shi,  L.,  Chen,  C.,  Wang,  Q.,  &  Boehm,  B.  "Maintenance  Effort  Estimation  for  Open  Source 
Software:  A  Systematic  Mapping  Study."  InProceedings,  ICSME  2016. 

Boehm,  B.,  Chen,  C.,  Srisopha,  K.,  &  Shi,  L.  2016.  "The  Key  Roles  of  Maintainability  in  an 
Ontology  for  System  Qualities."  In  Proceedings,  INCOSE  IS  2016. 

Hira,  Anandi,  Shreya  Sharma,  and  Barry  Boehm.  "Calibrating  COCOMO®  II  for  projects  with  high 
personnel  turnover."  Proceedings  of  the  International  Conference  on  Software  and  Systems 
Process.  ACM,  2016. 

Hira,  Anandi,  and  Barry  Boehm.  "Function  Point  Analysis  for  Software  Maintenance."  Proceedings 
of  the  10th  ACM/IEEE  International  Symposium  on  Empirical  Software  Engineering  and 
Measurement.  ACM,  2016. 

Hira,  Anandi,  and  Barry  Boehm.  "Using  Software  Non-Functional  Assessment  Process  to 
Complement  Function  Points  for  Software  Maintenance."  Proceedings  of  the  10th  ACM/IEEE 
International  Symposium  on  Empirical  Software  Engineering  and  Measurement.  ACM,  2016. 
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Hira,  Anandi,  and  Barry  Boehm.  "Combatting  Use  Case  Points'  Limitations  with  COCOMO®  and 
Relative  Difficulty."  Proceedings  of  the  23rd  Asia-Pacific  Software  Engineering  Conference.  IEEE, 
2016. 


Ross,  Adam  and  Rhodes,  Donna,  "Interactive  Model  Trading  for  Resilient  Systems  Decisions," 
Proceedings,  CSER  2017. 
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